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1.0  INTRODUCTION 


) 


1.1 


Objectives 


The  overall  objective  of  this  project  has  been  to  develop  a  prototype 
decision  aid  that  successfully  blends  two  features:  (1)  it  is  personalized 
in  the  sense  that  it  accommodates  individual  differences  in  beliefs, 
values,  preferred  methods  of  problem  structuring  and  analysis,  preferred 
methods  of  organizing  information  and  searching  a  database,  and  variations 
in  cognitive  "style,"  ranging  from  intuitive  to  analytical;  and  (2)  it  is 
prescriptive  in  the  sense  that  it  attempts  in  a  variety  of  ways  to  steer 
users  away  from  potential  biases  or  errors  commonly  found  in  decision 
making  and  associated  with  preferred  decision  making  strategies.  During 
Phase  I,  a  concept  for  such  an  aid  was  developed  and  demonstrated  in  the 
context  of  a  submarine  commander's  decision  as  to  when  to  launch  an  attack 
on  an  enemy  target.  This  decision  involves  a  difficult  tradeoff  between 
waiting  to  obtain  a  more  accurate  fire  control  solution,  and  increasing  own 
vulnerability  to  enemy  attack.  The  attack  planning  situation  was  rich 
enough  to  allow  incorporation  of  a  wide  range  of  personalizing  options 
selectable  by  individual  users,  as  well  as  advisory  prompts  calling  the 
user's  attention  to  potential  pitfalls  or  inconsistencies  and  alerting  him 
to  critical  events.  This  demonstration  prototype  aid  was  described  in  the 
Phase  I  Technical  Report  (Cohen,  et  al .  ,  1982). 


The  original  objectives  of  Phase  II  were  as  follows: 


(1)  Complete  computer  implementation  of  the  design  concept 
developed  in  Phase  I. 

(2)  Design,  implement,  and  test  enhancements  of  the  original 
concept,  by  increasing  the  degree  of  personalization  and  the 
scope  of  the  coverage  of  the  advisory  prompts. 

(3)  Design,  implement,  and  test  a  general-purpose  (non-testbed 
specific)  personalized  decision  aid. 

Early  in  Phase  II  it  was  decided,  in  conjunction  with  the  ON’R  Scientific 
Officer,  that  the  general-purpose  personalized  aid  would  take  the  form  of  a 
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system  for  evaluating  and  selecting  options  from  a  large  database.  In 
order  to  stimulate  potential  commercial  interest  in  the  development  of  the 
concept,  it  was  further  decided  that  the  database  subject  matter  for 
initial  application  of  the  system  should  be  a  domain  of  widespread 
interest,  and  the  problem  domain  of  personnel  selection  (from  a  large 
database  of  applicant  resumes)  was  chosen.  The  personalized  aid  itself, 
however,  including  the  analytic  model  and  interactive  features,  was  to  be 
generally  applicable  to  any  kind  of  option  evaluation  problem.  Finally,  in 
order  to  ensure  that  the  general-purpose  aid  had  the  highest  degree  of 
personalization  and  prescriptive  capability  possible,  it  was  decided  to 
focus  the  effort  involved  in  Objective  (2)  upon  the  general-purpose  aid 
rather  than  the  submarine  attack  planning  aid.  In  this  way,  the  end 
product  would  have  the  highest  probability  of  being  broadly  applicable  in  a 
wide  variety  of  decision-making  domains. 

1 . 2  Outline 

This  report  provides,  in  Section  2.0,  a  background  summary  of  the  research 
literature  describing  the  nature  of  commonly  found  cognitive  biases  in 
decision  making,  and  the  variations  in  problem-solving  strategies 
characterizing  different  users,  or  indeed  the  same  user  at  different  times. 
It  thus  provides  a  rationale  for  the  personalizing  and  prescriptive 
features.  Section  3.0  describes  the  major  differences  between  the  attack 
planning  and  the  personnel  selection  decision  problems.  Section  4.0 
describes  the  general-purpose  aid,  highlighting  the  personalizing  and 
prescriptive  features.  Section  5.0  describes  a  preliminary  experimental 
test  of  the  system,  and  Section  6.0  presents  conclusions. 


2 . 0  BACKGROUND 


2 . 1  General  Description  of  the  Decision  Process 

The  decision-making  process  can  be  conceptual ized  quite  generally  as 
consisting  of  a  specific  set  of  cognitive  tasks  (Figure  1).  First,  goals 
or  objectives  must  be  known  or  identified  (if  these  are  not  present,  there 
is  no  motivation  to  decide  or  act).  Secondly,  current  circumstances, 
insofar  as  they  are  relevant  to  the  achievement  of  a  goal,  are  assessed. 

If  a  discrepancy  is  perceived  between  goals  and  reality,  options  for  action 
are  generated.  If  more  than  one  option  is  available,  a  choice  will  be 
made . 

This  is  by  no  means  a  rigid  sequence:  the  process  is  usually  iterative 
(for  example,  revising  goals,  reassessing  the  situation,  or  generating  new 
options  when  the  choice  process  fails  to  turn  up  an  acceptable 
alternative);  and  steps  may  be  skipped  (when,  for  example,  the  appropriate 
action  is  known  based  on  past  experience  with  very  similar  situations) . 

But  the  basic  set  of  possibilities  is  as  shown,  at  least  in  many  of  the 
decision  contexts  we  have  considered,  and  some  such  framework  is  critical, 
we  believe,  for  identifying  the  specific  aspects  of  human  performance  where 
personalized  and  prescriptive  aiding  may  be  of  use. 

It  is  convenient  to  break  each  of  these  major  tasks  down  into  more 
specialized  cognitive  subtasks.  For  example,  situation  assessment  consists 
of  collecting  and  viewing  data  or  evidence,  deriving  inferences,  developing 
some  sense  of  confidence  in  the  conclusions,  and  continuing,  perhaps,  to 
draw  further  higher- level  inferences.  Again,  the  steps  may  be  iterative, 
may  be  combined,  or  may  be  skipped  altogether  by  some  decision  makers  in 
some  situations. 

(Note  that  the  term  "evidence"  is  quite  relative;  evidence  in  one  process 
may  be  the  highly  uncertain  conclusion  of  a  prior  analysis.) 


decision  making 


Assimilate 

Infer 

Assess 

Infer 

Generate 

Assess 

Assess 

Select 

Evidence 

Conclusions 

Quality  of 

Higher- Level 

Possible 

Uncertainty 

Values 

or  Reject 

from 

Evidence 

Conclusions 

Conclusions 

etc. 

Outcomes 
of  Options 

of  Outcomes 

of  Outcomes 

Options 

Figure  1:  Potential  Cognitive  Subtasks  in  the  Decision  Making  Process 


2 . 2  Basis  for  Prescriptive  Features 

During  the  past  10-12  years  a  substantial  amount  of  research  has  identified 
and  quantified  the  types  of  cognitive  biases  or  errors  commonly  made  during 
the  decision  process.  The  prescriptive  features  of  any  aid  must  be 
designed  to  prevent  or  counteract  these  types  of  biases,  to  the  extent 
possible . 

Each  of  the  cognitive  subtasks  identified  in  Figure  1  has  been  associated, 
at  least  in  laboratory  research,  with  characteristic  shortcomings  in 
reasoning.  Thus,  by  placing  recent  findings  in  cognitive  psychology  within 
this  framework,  we  may  derive  a  tentative  specification  of  the  types  of 
prescriptive  features  that  would  be  most  appropriate. 

The  following  summary  is  not  exhaustive;  it  is  meant  only  to  touch  on  some 
of  the  issues  that  bear  on  the  present  work.  Three  important  themes, 
however,  emerge:  (1)  Unaided  decision  processes  employ  simplifying 
heuristics  that  at  best  only  approximate  prescriptively  accepted  rules 
(e.g.,  Bayesian  probability  theory);  (2)  a  typical  effect  of  such 
heuristics  is  that  awareness  of  uncertainty  is  suppressed;  and  (3)  in  many 
instances,  biases  are  a  result  of  (otherwise  successful)  efforts  to  utilize 
natural  knowledge  structures  and  processes  of  reasoning. 

Assimilate  Evidence .  Patterns  of  information  search  in  laboratory  tasks 
tend  to  avoid  stringent  tests  of  favored  hypotheses  (Wason,  1960,  1981; 
Einhorn,  1980).  At  the  same  time,  there  is  a  tendency  to  seek  confirming 
evidence  of  an  already  well  -  supported  hypothesis,  rather  than  take  action 
or  consider  evidence  that  bears  on  other  issues  (Shaklee  and  Fischhoff, 

1982)  . 

Infer  Conclusions .  A  number  of  studies,  which  show  that  a  statistical 
model  of  a  person's  judgment  process  can  outperform  (in  accuracy)  that 
person's  own  judgments,  suggest  that  people  do  not  effectively  utilize  the 
information  available  to  them  in  inference  tasks  (Dawes,  1973;  Cohen, 

1982),  Other  laboratory  results  suggest  possible  causes. 


For  example. 


people  tend  to  ignore  later  evidence  th.it  contradicts  a  favored,  or 
earlier,  datum  and  to  double  count  redundant  evidence  (Schum  and  Martin, 


1981).  Also,  people  commonly  ignore  statistical,  or  "base  rate",  data  and 
overweight  unique  or  problem- spec  if ic  factors  (Kahneman  and  Tversky,  1972). 
Both  of  these  observations  suggest  the  predominance  in  natural  reasoning  of 
non - s t at i s t ica 1 ,  causal  models  (Johnson,  1489).  Results  can  be  distorted, 
and  overconfidence  can  occur,  when  false  analogies  between  the  system  and 
the  model  influence  conclusions.  When  people  do  attempt  to  make 
statistical  judgments,  moreover,  estimates  may  be  biased  by  the  ease  of 
recall  ( or  "availability")  of  a  particular  class  of  events  in  a  mental 
sampling  (Tversky  and  Kahneman,  1973). 

Assess  Quality  of  Conclusions.  A  number  of  studies  show  that  people 
consistently  overestimate  their  degree  of  certainty  regarding  predicted 
events  and  estimated  quantities,  even  in  areas  where  they  are  (rightfully) 
regarded  as  experts.  While  there  is  some  evidence  that  experts  (as  opposed 
to  college  sophomores)  are  less  susceptible  to  overconfidence 
(Lichtenstein,  Fischhoff,  and  Phillips,  1982),  other  research  indicates 
that  the  difference  between  expert  and  novice  is  slight  (Kadane  and 
Lichtenstein,  1982).  When  inference  proceeds  in  stages  (e.g.,  deriving  the 
probability  of  being  hit  by  enemy  fire  from  information  about  the  range  of 
a  threat,  which  is  derived  from  bearings  data),  people  often  simplify  the 
process  by  acting  as  if  conclusions  at  earlier  stages  (e.g.,  range)  were 
known  to  be  true,  rather  than  merely  inferred  (Schum,  DuCharme,  and 
DePitts,  19/3).  Similarly,  the  probability  of  a  detailed  hypothesis  or 
scenario  is  likely  to  he  judged  higher  than  the  probabilities  of  its 
components  (Tversky  and  Kahneman,  1983).  The  latter  effect  may  arise 
because  additional  details  increase  the  match  between  the  hypothesis  and 
the  userhi  mental  models  or  knowledge  structures  (Leddo,  Abe  Ison,  and 
Cross,  1984). 

Option  Generation.  People  segment  complex  opt  ions  into  "natural" 
components,  and  treat  the  elements  as  if  thev  w.  re  i  ndependi  n.t  choices , 
leading  to  subopt  imal  port  fol  i  os  (Tver.'  !  v  and  Kahneman,  I'AHl  )  .  There  is  a 
tendency  to  formulate  opt  ions  in  terms  of  i  ini  ed  i  a :  e  ,,ct  ions  that  span  onlv 


a  short  timeframe  rather  than  as  long-term  policies,  and  to  overlook,  as  a 
result,  the  cumulative  risk  of  pursuing  a  given  course  of  action  over  a 
long  period  of  time  (Slovic,  Fischhoff,  and  Lichtenstein,  1978). 

Individuals  differ  in  the  degree  to  which  they  consider  future  choices  in 
current  planning  (Streufert  and  Streufert,  1961)  and  in  the  number  of 
options  they  generate  (Driver  and  Monk,  1976).  Ingrained  ways  of  viewing  a 
problem  tend  to  hinder  the  generation  of  novel  and  creative  solutions 
(Pitz,  Sachs,  and  Heerboth,  1980). 

Generate  Possible  Outcomes  of  Options.  In  considering  what  might  happen  if 
a  particular  option  is  adopted,  people  are  subject  to  biases  based  on  their 
internal  causal  models,  as  well  as  biases  in  recall,  such  as  a  heightened 
tendency  to  remember  salient  events  or  events  that  occurred  very  late  or 
very  early  in  a  sequence. 

Assess  Uncertainty  of  Outcomes.  Some  of  the  biases  which  affect  situation 
assessment  may  also  occur  when  predictions  are  made  contingent  on  a 
particular  option.  Additional  pitfalls,  however,  include  the  effects  of 
"wishful  thinking"  (e.g.,  higher  probability  assessments  for  high  utility 
outcomes)  or  overcautiousness  (e.g.,  lower  assessments  for  high  utility 
outcomes).  According  to  Einhorn  and  Hogarth  (1984),  the  size  of  these 
effects  will  depend  on  the  degree  to  which  decision  makers  lack  confidence 
in  the  probability  estimates.  This,  in  turn,  may  depend  on  the  degree  to 
which  evidence  for  an  estimate  matches  the  type  of  evidence  represented  in 
user  knowledge  structures.  An  additional  set  of  biases  involves  distorted 
conceptions  of  randomness  in  everyday  judgment,  e.g.,  the  "gambler's 
fallacy"  where  a  sequence  of  similar  outcomes,  which  are  in  fact 
independent,  is  thought  to  increase  the  likelihood  of  a  different  outcome 
on  the  next  trial.  Fallacies  of  this  sort  may  be  inevitable  by-products  of 
powerful  top-down  or  expectancy-driven  processes  of  pattern  recognition 
(Lopes ,  1982) . 

Assess  Value  of  Outcomes .  Decision  makers  do  not  typically  consider  all 
the  potential  outcomes  of  an  action  together.  Rather,  outcomes  are  grouped 
into  "mental  accounts"  corresponding  to  natural  objects  or  causal 


relations,  and  choices  may  depend  critically  on  the  particular  grouping 
that  is  adopted  (Kahneman  and  Tversky,  1982).  An  additional  cognitive 
simplification  is  achieved  by  representing  an  outcome  in  causally  relevant 
terms,  by  the  difference  it  would  make  relative  to  some  reference  point. 
Decisions  may  be  significantly  affected  by  the  choice  of  reference  levels, 
since  the  same  outcome  may  be  regarded  as  a  gain  or  as  a  loss.  For 
example,  the  outcome  of  a  defensive  tactic  may  be  encoded  as  400  men  saved 
(relative  to  the  number  who  would  have  died  had  nothing  been  done)  or  as 
200  men  lost  (relative  to  the  status  quo).  An  important  finding  by 
Kahneman  and  Tversky  (1979)  is  that  decision  makers  are  more  likely  to  take 
risks  when  outcomes  are  represented  as  losses  than  when  they  are 
represented  as  gains. 

Select  an  Option.  Heuristic  procedures  may  be  adopted  which  reduce  the 
cognitive  effort  that  would  be  required  in  a  thorough  consideration  of 
every  option.  Such  heuristics  have  implications  for  the  way  decision 
makers  search  information.  In  Elimination  by  Aspects  (Tversky,  1972),  for 
example,  search  is  organized  by  evaluative  attributes.  Attributes  are 
considered  serially  in  order  of  importance;  options  falling  below  a  cut- 
point  on  an  attribute  are  eliminated  at  each  stage,  and  not  considered 
further.  In  this  strategy,  an  option  might  be  eliminated  for  missing  a 
cut-point  on  one  dimension  even  though  it  scores  very  highly  on  other 
dimensions.  Tradeoffs,  or  compensatory  relations  among  dimensions  are  thus 
not  considered.  In  another  heuristic  strategy,  called  "satisficing" 

(Simon,  1957;  Svenson,  1979},  information  search  is  organized  by  options. 
The  decision  maker  considers  a  sequence  of  options  until  he  finds  one  that 
clears  the  cut-points  he  has  selected  on  relevant  attributes.  Here  again 
compensatory  relationships  are  ignored.  Payne  (1981)  has  suggested  that 
these  information  search  strategies  may  correspond  to  the  way  decision 
makers  organize  knowledge. 

2 . 3  Basis  for  Personalized  Features 

How  are  the  users  of  decision  aids  likely  to  dill  or  in  their  approaches  to 
decision  making  and  problem  solving?  What  are  the  consequences  of  su  h 
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differences  for  success  in  task  performance?  And  how  should  aids  be 
personalized  so  as  to  enhance  both  user  acceptability  and  quality  of 
performance? 

We  consider,  briefly,  two  general  ways  in  which  decision  makers  have  been 
thought  to  differ  from  one  another: 

o  in  the  parameters  and  structure  of  a  prescriptive  model  based 

on  their  personal  beliefs  and  preferences;  and 

o  in  the  heuristic  strategies,  decision  processes,  and  cognitive 

styles  which  they  adopt  in  problem-solving. 

The  interplay  of  findings  from  these  areas  helps  define  the  potentialities 
and  limitations  of  personalized  decision  aiding. 

2.3.1  Individual  prescriptive  decision  models.  Ironically,  a  driving 
force  in  the  evolution  of  prescriptive  theories  of  decision  making  has  been 
the  need  to  accommodate  individual  differences.  An  objective  rule  for 
betting  in  games  of  chance,  maximization  of  expected  value,  applies  only 
where  probabilities  of  outcomes  can  be  mathematically  defined  (as  in 
rolling  dice)  and  where  the  desirability  of  outcomes  is  physically 
measurable  (e.g.,  by  money).  Generalizations  of  this  basic  rule  to 
situations  where  those  conditions  do  not  hold  have  led  to  the  modern 
technique  of  decision  analysis  (cf . ,  Edwards,  1934,  1961;  Raiffa,  1968; 
Brown,  Kahr,  and  Peterson,  1974).  Von  Neumann  and  Morgenstern  (1947) 
formalized  the  notion  of  a  subjective  dimension  of  value,  i.e.,  utility, 
and  extended  it  to  individual  preferences  among  probabilistic  states  of 
affairs.  De  Finetti  (1937/1964)  and  Savage  (1954)  developed  formal  systems 
for  the  quantification  of  an  individual's  "degree  of  belief",  or  subjective 
probability,  about  uncertain  propositions,  and  developed  axiomatic 
justif ications  for  the  merging  of  utilities  and  subjective  probabilities 
into  a  new  prescriptive  rule,  maximization  of  subjectively  expected 
utility.  More  recently,  rigorous  techniques  have  been  developed  for 
combining  subjective  preferences  with  respect  to  individual  components  of 
value  into  a  single  multiattribute  utility  measure  (e.g.,  Keeney  and 
Raiffa,  1976). 
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The  prescriptive  force  of  decision  analysis,  in  this  form,  is  not  to 
dictate  to  an  individual  in  any  absolute  sense  what  he  "ought"  to  do  or 
believe.  Rather,  it  indicates  what  choices  and  beliefs  are  logically 
consistent  with  other  preferences  and  beliefs  which  he  chooses  to  accept 
(cf . ,  French,  1979). 

These  elements  of  personalization  are  by  no  means  shared  by  all 
prescriptive  approaches.  Techniques  in  operations  research  (e.g., 
cost/benefit  analysis)  commonly  purport  to  be  "objective"  and  "value  free" 
(Watson,  1981).  The  approach  to  decision  analysis  described  above, 
however,  has  two  important  implications  for  personalized  aids: 

(1)  Decision-analytic  aids  do  not  address  only  the  part  of  a  problem  that 
can  be  objectively  measured.  Actual  decisions  nearly  always  involve  a 
number  of  "soft  factors"  (e.g.,  uncertainty  about  the  intentions  of  a 
business  competitor  or  of  a  military  foe;  the  relative  importance  of 
different  objectives,  like  money  and  prestige).  The  decision  maker's  own 
experience  may  be  the  only  source  of  relevant  information  in  these  matters, 
while  an  exclusively  "factual"  approach  could  be  fatally  incomplete.  Aids 
which  combine  subjective  and  objective  inputs  must  accommodate  individual 
differences  among  users  in  assessments  of  uncertain  states  of  affairs, 
attitudes  toward  risk,  and  tradeoffs  among  competing  objectives. 

(2)  The  second  point  is  equally  important,  though  far  less  widely 
recognized.  Just  as  it  does  not  prescribe  inputs,  decision  theory 
constrains,  but  does  not  dictate  problem  structure.  Typically,  there  is 
more  than  one  way  to  express  the  probability  of  a  hypothesis  in  terms  of 
probabilities  for  other  propositions;  and  there  are  multiple  decompositions 
of  the  utility  of  an  option  into  preferences  for  separate  attributes.  A 
good  structure  for  a  particular  decision  maker  breaks  the  problem  down  into 
components  about  which  that  decision  maker  has  either  objective  data  or 
personal  experience.  Individuals  might  benefit  differently  from  different 
analyses  of  the  same  problem. 


In  particular,  it  has  been  suggested  that  experts  differ  from  novices  in 
their  capability  to  individually  recognize  a  very  large  number  of  different 
problem  situations  (De  Groot,  1965;  Chase  and  Simon,  1973).  Klein  (1980) 
argues  that  experts  tend  to  reason  holistically,  by  analogy  with  previous 
similar  experiences,  rather  than  by  explicit  analysis  and  computation. 

Klein  warns  that  imposition  of  analytical  models  may  actually  impair  expert 
performance.  In  terms  of  decision  theory,  however,  this  distinction 
between  experts  and  novices  is  accommodated  by  the  notion  of  personalized 
problem  structures.  The  expert  might  produce  quite  creditable  holistic 
judgments  of  problem  components  which  he  has  "seen  before"  but  which  a  less 
experienced  individual  would  need  to  analyze  into  more  familiar  elements. 
(Nonetheless,  experts  too  are  subject  to  error- -particularly  when  a  problem 
which  appears  familiar  has  novel  aspects;  cf . ,  Sage,  1981.  Experts  may 
benefit  from  analysis  of  such  novel  components.)  The  implication  is  that 
if  decision  aids  are  to  exploit  the  capabilities  of  each  potential  user,  a 
variety  of  models,  with  different  functions  and  at  different  levels  of 
aggregation,  should  be  made  available  (cf . ,  Strub  and  Levit,  1974). 

2.3.2  Individual  strategies  in  inference  and  choice.  Prescriptive 
decision  theory  does  not  provide  a  description  of  actual  performance, 
either  in  probabilistic  reasoning  or  in  the  evaluation  of  actions  (cf . , 
Einhorn  and  Hogarth,  1981).  Recent  research  in  cognitive  psychology  has 
shed  light  on  the  internal  processes  and  structures  which  people  employ  in 
such  tasks,  and  how  they  differ. 

One  line  of  research  has  explored  the  strategies  people  use  in  choosing 
among  actions.  Prescriptive  theory  requires  that  a  single  score  for  each 
option  (its  expected  utility)  be  derived,  which  integrates  all  the 
available  information  about  that  option:  i.e.,  its  score  on  each  of  a  set 
of  attributes,  or  the  probabilities  and  utilities  of  its  possible  outcomes. 
Several  descriptive  models  of  choice  behavior  have  been  proposed,  however, 
which  involve  more  partial  samplings  of  the  available  data  (e.g.,  Payne, 
1973;  Svenson,  1979). 


In  Tversky's  (1972)  Elimination-by -Aspects  (EBA),  for  example,  (as 
described  in  Section  2.2  above),  the  decision  maker  sequentially  considers 
each  attribute,  establishes  a  threshold,  and  eliminates  all  options  that  do 
not  score  at  or  above  the  threshold  on  that  attribute.  In  the  decision 
strategy  called  "satisficing"  (Simon,  1957;  Svenson,  1979),  the  decision 
maker  adopts  a  conjunctive  criterion  involving  cutoffs  on  one  or  more 
dimensions,  and  compares  successive  options  to  the  criterion  until  he  finds 
one  that  is  acceptable,  whereupon  he  stops.  These  different  decision 
strategies  have  different  implications  for  the  order  in  which  people  e]ect 
to  receive  information  (Payne,  1973,  1976).  Some  strategies  imply  a  search 
organized  by  options,  others  a  search  organized  by  attributes. 

Individual  decision  makers  vary  in  the  decision  strategies  which  are 
reflected  in  their  information- seeking  behavior  and  in  their  verbal 
protocols  (Payne,  1976;  Russo  and  Dosher,  1981).  But  little  work  has  been 
done  to  discover  whether  these  individual  differences  are  consistent  across 
time  and  tasks  (Svenson,  1979);  instead,  emphasis  has  been  on  the  role  of 
task  variables.  For  example,  when  there  are  a  large  number  of  choice 
options,  decision  makers  tend  to  select  routines  like  EBA  which  quickly 
eliminate  some  options  by  more  approximate  methods.  They  may  then  switch 
over  to  routines  which  integrate  all  the  available  information  about  the 
remaining  options  (Payne,  1976;  Wright  and  Barbour,  1977). 

Cognitive  style  has  been  regarded  as  a  relatively  invariant,  abstract 
feature  of  a  decision  maker's  approach  to  information  across  a  variety  of 
tasks  (cf . ,  Sage,  1981;  Libby  and  Lewis,  1977).  Perhaps  the  most  common 
differentiation  made  in  this  literature  is  represented  by  a  related  cluster 
of  distinctions  between  "analytic"  and  "heuristic"  (Huysman,  1970;  Mock, 
Estrin,  and  Vasarhelyi,  1972),  "abstract"  and  "concrete"  (Schroder,  Driver, 
and  Streufert,  1967;  Sage,  1981),  "systematic"  and  "intuitive"  (Bariff  and 
Lusk,  1977;  McKenney  and  Keen,  1974),  and  "scientific"  and  "managerial" 
decision  makers.  The  common  thread  is  a  distinction  between  preference  for 
formal,  explicit  analysis,  breaking  a  problem  down  into  elements,  and  an 
approach  based  on  global  intuition,  trial  and  error,  or  "common  sense". 
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Unfortunately,  there  is  little  evidence  establishing  a  relationship  between 
these  categories  (based  on  self -descriptions)  and  actual  information¬ 
seeking  behavior  (Zmud,  1979;  Keen,  undated).  It  has  been  found  that 
systematics  generally  take  more  time  and  do  better  in  decision  problems 
than  heuristics  (e.g.,  Mock  et  al . ,  1972).  Other  results,  however,  have 
been  inconsistent,  showing  that  systematics  prefer  more  information  or 
less  information  and  prefer  either  aggregated  or  raw  data  as  compared  to 
heuristics  (cf . ,  Libby  and  Lewis,  1977;  Zmud,  1979).  McKenney  (quoted  in 
Mock  et  al . ,  1972)  states  that  the  propensity  to  be  analytical  increases 
with  task  familiarity;  Klein  (1980)  and  Sage  (1981)  suggest  that  experts 
will  be  more  intuitive. 

A  second  problem  in  this  literature  is  the  failure  to  validate  the  claim 
that  cognitive  styles  are  task  invariant.  Studies  which  have  attempted  to 
do  so  have  produced  disappointing  results  (cf.,  Libby  and  Lewis,  1977),  and 
recent  reviews  (Libby  and  Lewis,  1977;  Sage,  1981)  have  shifted  emphasis 
toward  the  influence  of  task  features  on  decision  styles  adopted  by  the 
same  individual  at  different  times.  Indeed,  Hammond,  et  al . ,  (1984)  have 
shown  that  not  only  does  the  nature  of  the  task  influence  decision  style, 
but  that  particular  ways  of  presenting  task-related  information  can  cause  a 
shift  toward  analytic  or  intuitive  methods. 

In  a  few  cases,  "cognitive  styles"  have  been  defined  in  relation  to  actual 
cognitive  behavior.  Thus,  Driver  and  Mock  (1976)  defined  four  styles  by 
reference  to  two  fairly  specific  processing  dimensions:  amount  of 
information  used  and  degree  of  focus.  The  latter  refers  to  a  tendency  to 
consider  only  one  solution,  model,  or  option  versos'  a  tendency  to  entertain 
multiple  possibilities.  Streufert  and  Streufert  (1481a)  present  criteria 
for  "integrative"  decision-making  styles  in  terms  of  the  number  of,  and 
length  of  time  between,  information  requests  and  decisions  based  upon  them. 
Streufert  and  Streufert  (198?b)  report  that  integrative  decision  making 
decreases  with  decision  urgency,  but  is  an  invert ed-U - shaped  function  of 
the  amount  of  information  available. 


2 . A  Implications  for  a  Personalized  and  Prescriptive  Aid 


Descriptive  work  on  human  inference  and  decision  processes  has  implications 
for  both  the  personal  and  prescriptive  aspects  of  decision  aiding. 

2.A.1  Personalization  and  efficient  flexibility.  "Flexibility"  in  and  of 
itself  is  not  a  sufficient  objective  in  system  design.  It  is  possible  to 
make  each  of  a  vast  number  of  logically  possible  information  acquisition 
strategies  equally  easy,  by  allowing  the  user  to  indicate  what  he  wants 
item-by- item .  But  such  a  system  does  not  really  facilitate  the  selection 
of  strategies  as  such;  to  deal  explicitly  with  all  possible  search  orders 
would  be  beyond  the  time  and  capabilities  of  both  user  and  device.  The 
objective  of  personalization  is  to  delimit  the  subset  of  strategies  which 
an  individual  is  most  likely  to  prefer.  Decision  aids  may  then  be  tuned  to 
facilitate  explicit  selection  from  this  smaller  group  of  strategies,  while 
still  affording  the  general  "flexibility"  of  an  arbitrary  item-by- item 
search  sequence.  Such  aids  are  efficiently  flexible  in  their 
responsiveness  to  likely  user  needs. 

The  most  natural  way  to  acquire  and  process  information  can  vary  as  a 
function  of  the  individual  and  the  task.  Several  such  forms  of  variation 
seem  to  occur  frequently  enough  in  performance  to  justify  an  aid  design 
which  facilitates  their  employment: 

o  search  organized  by  options  or  by  attributes, 

o  decision  rules  based  on  cutoffs  or  tradeoffs, 

o  level  of  aggregation  of  informat ion . 

In  addition,  it  seems  desirable  that  an  aid  facilitate  differences 
involving : 

o 

o 


o 


focus  on  one  or  many  options, 
desired  amount  of  information,  and 

time  into  the  future  over  which  planning  takes  place. 


There  is  little  evidence  that  particular  individuals  are  consistent  across 
tasks  in  these  preferences,  and  some  indication  that  they  are  not.  In  the 
case  of  gross  categories  like  "intuitive"  and  "analytic”,  moreover,  there 
is  no  reliable  mapping  of  traits  onto  system  design  features  and  certainly 
no  indication  of  how  different  traits  interact  (cf.,  Huber,  1982). 

2.4.2  Prescriptive  aiding.  The  danger  inherent  in  complete  flexibility 
for  the  decision  maker  is,  of  course,  the  high  likelihood  that  one  or  miore 
of  the  common  cognitive  biases  described  in  Section  2.2  will  result.  As  a 
safeguard  against  this,  two  types  of  prescriptive  aids  may  be  introduced: 
channeling  and  advisory  prompting.  The  difference  between  them  is  largely 
one  of  tactics.  Channeling  is  implicit  and  proactive,  in  that  it 
encourages  users,  in  advance,  to  adopt  variants  of  their  own  preferred 
strategies  which  are  less  susceptible  to  biases  and  fallacies  of  judgment, 
by  structuring  the  problem  in  such  a  way  that  those  variants  become  natural 
and  simple  to  execute.  By  contrast,  advisory  prompting  is  explicit  and 
reactive,  in  that  ‘he  system  monitors  the  tasks  performed  by  the  human, 
identifies  steps  taken  that  are  likely  to  lead  to  error,  and  prompts  for 
the  addition  of  procedures  that  mesh  with  the  preferred  strategy  but  would 
minimize  the  possibility  of  error;  it  also  monitors  tasks  performed  by  the 
computer  and  prompts  where  a  human  contribution  might  improve  results. 

Thus,  in  advisory  prompting  the  computer  senses  weaknesses  in  a  line  of 
reasoning,  whether  its  own  or  the  user's,  and  offers  help.  Both  channeling 
and  advisory  prompting  may  be  viewed  as  examples  of  low-level  expert 
systems . 

Some  examples  of  how  the  prescriptive  aiding  techniques  of  channeling  and 
advisory  prompting  can  be  blended  into  a  personalized  system  are  given 
be  low . 

While  users  should  be  able  to  organize  displays  around  a  variety  of 
m.  aningful  user-designated  objects,  the  aid  should  facilitate  the  use  of 
decision- related  objects  for  this  purpose.  For  example,  channeling  can 
facilitate  clustering  of  options  by  their  performance  on  a  selected 
evaluative  criterion.  When  an  intermediate  result  or  conclusion  is 


uncertain,  the  sources  of  its  uncertainty  should  be  explicitly  indicated. 
Evidence  for  a  result  should  be  available  for  display  along  with  the 
result.  Inferential  relationships  in  the  database  can  be  "mapped"  by 
menus,  which  permit  tracing  a  process  of  reasoning  from  its  sources  of 
evidence  to  its  final  conclusion. 

Research  in  cognitive  psychology  suggests  that  humans  tend  to  seek 
additional  confirming  evidence  for  a  favored  hypothesis.  An  advisory 
prompt  might  monitor  a  user's  pattern  of  information  requests,  examine  its 
own  model  of  the  problem  in  order  to  draw  inferences  about  the  hypotheses 
the  user  has  in  mind,  and  prompt  the  user  if  evidence  or  hypotheses  exist 
which  the  user  has  failed  to  consider  but  which  may  have  an  impact  on  his 
conclusions . 

Humans  often  find  it  difficult  to  assess  the  overall  credibility  of  a 
conclusion  based  on  several  steps  of  reasoning;  they  simplify  by  ignoring 
the  uncertainty  at  early  stages.  Prompts  might  warn  users,  when  they 
appear  to  be  acting  as  if  a  particular  hypothesis  were  known  to  be  true, 
that  a  number  of  stages  of  uncertainty  must  be  kept  in  mind.  The  same  type 
of  caution  might  be  appropriate  when  a  compound,  or  conjunctive,  hypothesis 
is  being  considered. 

The  user  might  be  notified  when  two  information  sources,  both  of  which  are 
regarded  as  credible,  have  contradicted  one  another.  He  might  then  choose 
to  readjust  one  or  both  credibility  assessments  downward.  An  advisory 
prompt  might  notify  him  on  future  occasions  when  either  of  the  (partially) 
discredited  sources  is  involved  in  an  important  conclusion. 

While  the  aid  should  permit  user  adjustment  of  any  meaningful  values 
employed  in  the  database,  channeling  should  selectively  facilitate 
adjustment  of  values  about  which  users  are  likely  to  have  information  not 
available  to  the  computer.  Values  to  be  adjusted  could  be  decomposed  by 
channeling  into  parameters  about  which  users  are  likely  to  have  reliable 
intuitions.  Automatically  computed  values  could  be  displayed  as  a 
reference,  so  users  can  focus  on  the  appropriate  direction  and  magnitude  of 


the  adjustment  (based  on  the  new  evidence)  and  not  have  to  integrate  all 
the  evidence  to  come  up  with  an  absolute  value. 

Humans  tend  to  combine  evidence  by  a  process  that  is  more  like  averaging 
than  like  proper  Bayesian  inference.  When  adjustments  fit  an  averaging 
pattern,  advisory  prompts  might  remind  subjects  to  consider  what  conclusion 
a  new  bit  of  evidence  favors,  before  performing  an  adjustment. 

Users  could  be  prompted  when  information  they  possess  may  be  of  significant 
value,  i.e.,  when  (1)  there  is  incompleteness  of  evidence  or  a  conflict 
among  lines  of  reasoning  in  the  computer  model  of  the  problem;  (2)  the  user 
has  potential  access  to  relevant  information;  and  (3)  the  result  is 
expected  to  have  an  impact  on  choices  among  actions  and  ultimate  payoffs. 

Channeling  could  facilitate  relatively  long  time  horizons  for  planning 
(e.g.,  by  displaying  appropriate  scaling).  Simultaneous  specification  of 
all  components  of  a  complex  option  could  be  facilitated  by  channeling. 
Displays  should  permit  generation  and  simultaneous  comparison  of  multiple 
options.  Channeling  should  facilitate  generation  of  options  which  include 
future  choices  or  contingencies. 

Short-range  planning  might  be  more  appropriate  in  some  situations  (e.g., 
where  feedback  is  continuous  and  mistakes  can  be  easily  and  quickly 
corrected) ,  while  long-range  planning  would  be  more  suitable  in  others 
(e.g.,  where  a  risk  appears  small  unless  it  is  considered  cumulatively  over 
the  long  run) .  Advisory  prompts  might  recommend  that  the  user  consider  a 
shift  in  the  time  horizon  under  appropriate  circumstances . 

Users  should  be  prompted  if  they  have  generated  and  evaluated  a  complex 
option  piece-by-piece  and  if  overall  optimality  would  be  significantly 
improved  by  considering  the  option  as  a  whole. 

The  user  should  be  prompted  if  only  one  option  has  been  considered,  but 
another  option  exists  which  is  superior  on  at  least  one  dimension. 
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The  user  should  be  prompted  if  contingency  plans  have  not  been  incorporated 
in  an  option,  but  significant  new  information  is  likely  to  become  available 
during  its  execution. 

Channeling  could  draw  the  user's  attention  to  tradeoffs  between  different 
evaluative  dimensions  by  displaying  scores  for  an  option  on  more  than  one 
dimension  concurrently  (e.g.,  costs  and  benefits).  The  aid's  action 
recommendations  should  be  explained  by  itemizing  how  options  differ  on  all 
significant  dimensions.  Channeling  should  encode  and  display  outcomes  in 
terms  of  more  than  one  reference  point  (e.g.,  assets  lost,  assets  saved). 

Humans  tend  to  employ  simplified  choice  schemes  that  disregard  tradeoffs. 

An  advisory  prompt  might  notify  the  user  when  he  has  eliminated  an  option 
because  it  fails  to  achieve  a  specified  level  on  a  favored  evaluative 
dimension,  if  that  option  has  significant  advantages  on  other  dimensions. 
The  user  might  be  told  how  much  stretching  of  his  specified  criterion  is 
required  to  readmit  the  rejected  option. 

An  advisory  prompt  should  occur  when  a  user  entertains  an  option  which  is 
dominated  (inferior  or  tied  on  all  dimensions)  by  other  options. 


While  users  should  be  free  to  designate  any  item  or  variable  in  the 
database  as  a  criterion  for  alerting,  alerts  should  also  occur  on  a 
prescriptive  basis.  Users  should  be  prompted  when  events  occur  or  facts 
are  learned  which  have  high  impact  within  an  appropriate  prescriptive 
model,  e.g.,  which  disconfirm  previously  we  1 1 - supported  inferential 
hypotheses  or  which  significantly  affect  choices  among  actions. 


The  distinction  between  channeling  and  advisory  prompting  is  based  to  some 
extent  on  the  source  of  the  potential  bias  or  error.  Using  the  terminology 
of  behavioral  decision  theory,  if  "base  rate  data"  (i.e.,  a  body  of 
research  findings)  suggests  that  errors  commonly  result  from  certain  ways 
of  representing  a  problem,  organizing  the  variables,  assessing  uncertainty, 
evaluating  outcomes  and  making  choices,  channeling  can  be  built  into  the 
aid  to  increase  the  chances  that  the  user  will  adopt  amended  versions  of 
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these  procedures  that  avoid  all  or  most  of  the  errors.  On  the  other  hand, 
if  "individuating  data"  (i.e.,  the  actual  procedures  and  judgments  of  the 
specific  user)  appear  to  be  moving  the  user  into  less  -  than- opt imal  choices 
advisory  prompting  can  explicitly  point  this  out  and  suggest  alternatives 
that  deviate  minimally  from  the  user's  originally  preferred  strategy 
(again,  preserving  the  user’s  freedom  to  ignore  the  advice).  In  this  way, 
the  flexibility  provided  by  personalization  of  an  aid  can  be  tempered  by 
the-  prescriptive  techniques  of  channeling  and  prompting  without  imposing 
undesired  constraints  on  tlie  user. 


3.0  INTRODUCTION  TO  NEW  DECISION  MODEL 


As  indicated  in  Section  1.1  the  prototype  aid  serving  as  a  context  for 
this  work  was  changed  from  one  dealing  with  a  submarine  commander's 
decision  about  when  to  launch  an  attack  on  an  enemy  target,  to  one  dealing 
with  a  personnel  manager's  selection  of  a  new  hire  from  a  number  of 
applicants.  Aside  from  the  obvious  difference  between  the  two,  namely, 
that  the  first  deals  with  military  tactics  while  the  second  deals  with  a 
broader  civilian  (as  well  as  military  support)  decision  situation,  there 
are  other,  more  fundamental  differences  between  the  two  types  of  decisions 
that  should  be  pointed  out  before  the  new  system  is  described  in  detail. 

3 . 1  Decree  of  Model  Generality 

s 

The  most  significant  difference  is  that  the  new  decision  problem  is  one 
that  lends  itself  to  a  much  more  generic  model,  applicable  to  a  wider 
variety  of  situations.  The  submarine  attack  model  is  typical  of  military 
tactical  situations  in  which  the  decision  maker  is  faced  with  a  clear 
tradeoff  between  firing  early  or  waiting  for  more  information  (which  may 
increase  the  probability  of  his  success  but  simultaneously  decrease  his  own 
survival  probability).  This  type  of  tradeoff  occurs  in  many  military 
tactical  situations,  but  the  temporal  pace  of  the  action  and  the  factors 
that  must  be  built  into  the  model  vary  considerably .  These  tactical  models 
must  be  largely  tailored  to  specific  situations,  hence  their  generic 
features  are  severely  limited. 

The  personnel  selection  problem,  on  the  other  hand,  is  typical  of  a  wide 
variety  of  decisions  that  involve  multiple  evaluative  criteria  or 
objectives.  Thus,  if  properly  constructed,  the  system  can  be  made 
applicable  to  other  decisions  such  as  choices  among  political  candidates, 
policy  options,  investment  portfolios,  R&D  programs,  military  plans,  and 
many  others.  This  generality  can  be  accomplished  )>v  keeping  the  modular 
process  programs  entirely  separate  from  the  database.  Thus,  there  c.«n  be 
modules  that  allow  the  user  to  build  his  own  preference  model ,  to 
accomplish  word  processing  functions  while  building  t  he  model ,  to  se..i eh 


the  database  in  various  ways,  to  organize  the  data  in  various  ways  for 
analysis  and  display,  to  enter  new  data  or  be  alerted  when  new  data  are 
entered  by  someone  else,  and  to  provide  channeling  and  advisory  prompting 
prescriptive  aids- -and  these  modular  programs  can  operate  on  any  type  of 
data  that  are  germane  to  the  specific  decision  problem. 

3 . 2  Real  Time  and  Stress 

Another  important  difference  between  the  two  decision  situations  is  that  in 
the  submarine  attack  decision,  real  time  plays  a  central  role  in  the  way 
the  problem  develops,  and  in  the  decision  itself,  and  contributes  to  the 
build-up  of  stress  in  the  situation.  In  the  personnel  selection  decision 
on  the  other  hand,  although  there  may  be  a  real  time  deadline  for  the 
decision,  the  process  itself  is  largely  self-paced,  under  the  control  of 
the  decision  maker,  and  stress  plays  a  negligible  role  in  the  situation 
(although  there  may  be  stressful  components  in  other  applications  of  this 
generic  model) .  In  the  submarine  attack  problem,  the  firing  decision 
evolves  over  a  period  of  time  during  which  (in  most  cases)  more  and  more 
information  becomes  available,  and  information  seeking  is  always  an 
important  option  to  be  considered.  In  the  personnel  selection  decision, 
although  it  is  sometimes  possible  to  obtain  additional  information,  the 
emphasis  is  on  evaluating  existing  data  in  a  variety  of  ways  in  order  to 
select  an  option. 

In  the  submarine  case,  once  an  action  has  been  selected  and  performed,  the 
problem  essentially  is  re-set  and  may  begin  again  later  with  a  new  target. 
In  the  personnel  selection  case,  the  problem  often  continues  after  a  choice 
is  made,  since  multiple  choices  are  typically  possible.  Here,  subsequent 
choices  may  be  made  from  among  fewer  options,  or  new  options  (and  new  data) 
may  be  introduced  into  the  decision  situation.  Thus,  in  this  respect  the 
personnel  selection  decision  offers  a  richer  set  of  conditions  under  which 
the  decision  process  may  be  examined. 


3 . 3  Inference  vs  Choice 

Although  it  is  often  difficult  to  separate  the  two  components  of  inference 
and  choice  in  real-life  decisions,  there  are  certain  key  distinctions 
between  them.  Inference  is  concerned  mainly  with  assessment  of  evidence 
related  to  the  relative  likelihoods  of  various  hypotheses  being  true,  and 
the  judgments  being  made  are  usually  in  the  form  of  probabilities. 

Although  these  probabilities  usually  enter  into  the  choice  process,  the 
distinctive  feature  of  choice  is  the  assessment  of  preferences  regarding 
the  various  options  available,  or  the  predicted  consequences  or  outcomes  of 
these  options.  In  the  submarine  attack  decision,  the  more  significant 
component  was  that  of  inference  (about  enemy  position  and  intent),  while  in 
the  personnel  selection  decision,  the  more  crucial  component  is  that  of 
preference  assessment. 


3 • 4  Availability  of  Objective  Data 


In  the  submarine  context,  there  is  little  or  no  opportunity  for  the  user  to 
assess  and  adjust  the  validity  of  his  decision  model  by  reference  to  on¬ 
going  results  of  applying  the  model.  The  personnel  selection  decision,  on 
the  other  hand,  is  typically  much  richer  in  terms  of  its  database  of  on¬ 
going  results.  The  database  can  include  data  on  the  characteristics  and 
performance  of  current  and  past  employees  (in  addition  to  applicant 
characteristics) ,  so  that  the  user  can  in  fact  test  his  model  against 
previous  data  if  he  desires.  As  a  result  of  this  feature,  the  personnel 
selection  application  offers  a  large  set  of  opportunities  for  individual 
variations  in  data  organization  and  display.  Users  may  change  the  features 
being  considered,  their  importance,  the  sequence  in  which  they  are 
examined,  and  the  way  they  are  displayed,  as  well  as  the  level  of  detail  at 
which  the  analysis  is  conducted.  As  pointed  out  in  Section  2.4,  safeguards 
in  the  form  of  prescriptive  aids  must  be  provided  to  minimize  the 
judgmental  biases  that  could  emerge  as  a  result  of  this  flexibility,  but 
the  overall  aims  of  the  project  are  well  served  in  the  context  of  the  new 
application  area. 
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4.0  SYSTEM  DESCRIPTION 


4 . I  High  Level  System  Design 

The  conceptual  design  of  the  personalized  evaluation  system  has  been 
organized  around  two  closely  related  concerns:  (1)  Users  of  a  large 
database  of  options  may  differ  in  the  extent  to  which  they  know  what  they 
want  or  do  not  want,  in  the  naturalness  with  which  they  make  holistic  vs. 
analytic  judgments  of  preference,  in  the  decision  rules  and  information 
search  strategies  they  favor,  and  in  the  extent  to  which  they  seek  to 
validate  choices  by  examining  low-level  evidence.  (2)  In  the  very 
experience  of  examining  and  evaluating  options,  user  preferences  may  grow 
increasingly  determinate,  with  a  corresponding  shift  in  decision  and 
information  search  strategies.  As  a  consequence  the  system  is  designed  to 
be  personalized,  that  is,  to  conform  to  a  user's  preferred  cognitive  style. 
The  user  of  such  a  personalized  aid  benefits  from  not  having  to  force 
his/her  thinking  into  an  uncomfortable  mold,  but  at  the  cost  of  relying  on 
a  strategy  that,  although  familiar  and  comfortable,  may  be  suboptimal.  To 
guard  against  this  risk,  the  aid  is  also  designed  to  be  prescriptive ,  to 
warn  the  user  of  information  that  may  have  been  ignored,  or  errors  in 
judgment  that  may  have  occurred. 

The  system  is  built  upon  a  set  of  elementary  modules,  consisting  of  a 
knowledge  module,  four  cognitive  interface  modules  that  process  and  make 
changes  in  the  information  stored  in  the  knowledge  module,  and  a  fifth 
cognitive  interface  module  whose  function  is  to  alert  the  user  when  actions 
taken  via  the  other  modules  may  differ  significantly  from  some  normative 
benchmark . 

The  experienced  user  can  access  these  modules  directly,  via  a  user-friendly 
interface  that  features  menu-driven,  mouse-controlled  graphical  displays. 
Users  may  also  wish  to  make  use  of  guides,  whose  purpose  is  to  step  the 
user  through  the  elementary  modules  in  accordance  with  one  of  a  set  of 
available  strategies  corresponding  to  different  cognitive  styles. 


4.1.1  Overview  of  elementary  modules.  A  set  of  six  basic  modules, 
interconnected  as  shown  in  Figure  2,  form  the  basis  of  a  generic  decision 
aid.  Depending  on  specific  context,  the  importance  attached  to  the 
functions  of  the  modules  may  vary.  Our  focus  in  this  report  is  on  an 
evaluation  aid  (specifically,  for  personnel  selection).  This  overview 
describes  the  generic  functions  of  each  module;  the  next  section  describes 
how  each  module  is  implemented  in  the  specific  context  of  personnel 
selection,  and  gives  examples. 

1.  Knowledge  Module  -  This  module  encodes  the  system's  knowledge  about 
the  decision  context.  Knowledge  includes  specific  information  about 
the  decision  problem  (e.g.,  options  and  facts  about  options),  general 
knowledge  about  the  problem  domain  (e.g.,  preferences  among 
evaluative  criteria) ,  procedural  knowledge  about  how  the  decision 
process  is  to  be  carried  out,  and  knowledge  about  the  current  status 
of  the  decision  process.  This  knowledge  base  serves  as  input  for, 
and  is  altered  by,  the  cognitive  interface  modules. 

2.  Adjust  Module  -  This  cognitive  interface  module  allows  the  user  to 
add  to  or  change  the  system's  database  of  specific  problem  knowledge 
and  general  knowledge  about  the  problem  domain.  The  user  can  also 
create  or  alter  procedural  knowledge,  i.e.,  knowledge  about  how  the 
database  is  to  be  manipulated.  Inputs  may  be  specified  in  different 
ways  and  at  varying  levels  of  "fuzziness,"  depending  on  the  user's 
preference . 

3.  Decide  Module  -  This  cognitive  interface  module  allows  the  user  to 
apply  the  system's  procedural  knowledge  to  the  system's  database. 

The  user  may  specify  one  of  several  decision  making  strategies  or 
models  (e.g.,  multiattribute  utility  evaluation,  elimination  by 
aspects),  corresponding  to  different  cognitive  styles,  for  database 
manipulation  and  choice.  Evaluation  proceeds  to  the  degree  permitted 
by  the  level  of  specificity  of  user  inputs,  and  whatever  implications 
can  be  drawn  are  displayed. 

4.  Select  Module  -  Using  this  cognitive  interface  module,  the  user  may 
select  a  subproblem  on  which  to  focus,  a  subset  of  information  to  be 
displayed,  or  a  subcategory  of  the  database  to  be  adjusted. 


5.  Alert  Module  -  This  cognitive  interface  module  prompts  the  user  when 
events  occur  or  facts  are  learned  which  may  cause  significant  changes 
in  user  decisions.  This  function  is  most  important  in  real-time 
decision  aids,  when  rapid  assimilation  and  incorporation  of  incoming 
information  is  essential. 
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Figure  2:  Basic  Modules  of  Personalized  Decision  Aid 
(as  applied  to  Personnel  Selection  Aid) 
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Advisory  Module  -  This  cognitive  interface  module  prompts  the  user 
when  he  or  she  appears  to  be  using  a  strategy  or  a  user-computer  task 
allocation  scheme  which  may  be  suboptimal  according  to  some  normative 
benchmark . 

4.1.2  The  Guides .  A  guide,  as  its  name  implies,  has  the  function  of 

directing  the  user  through  the  basic  functions  in  accordance  with  a  given 
decision  strategy.  Thus,  each  guide  is  tailored  to  a  particular  strategy, 
and  the  cognitive  style  of  the  user  determines  which  guide  or  guides  (s)he 

chooses  to  invoke.  The  guides  will  in  general  be  adaptive,  enabling  the 

user  to  observe  the  implications  of  certain  inputs  and,  if  necessary,  cycle 
back  to  change  them.  Each  strategy  is  subject  to  its  own  set  of  possible 

biases,  and  an  important  function  of  the  guide  is  to  inform  the  user  when 

the  possibility  of  normative  violations  occurs. 

4 . 2  Prototype  Personnel  Evaluation  System:  Elementary  Modules 

As  discussed  in  Section  3.0,  the  personnel  selection  context  has  several 
distinguishing  characteristics  which  affect  the  design  of  a  decision  aid. 
The  aid  is  focused  on  choice  as  distinguished  from  inference.  In  addition, 
time  stress  and  the  incorporation  of  uncertainty  are  of  secondary 
importance,  while  data  organization  and  display  are  of  prime  importance. 

In  this  section,  we  discuss  both  the  elementary  modules  and  the  guides  as 
they  apply  to  the  aid  under  development.  The  aid  is  generic  in  that  it  can 
be  used  in  any  problem  domain  sharing  the  above  characteristics. 

4.2.1  Knowledge  module.  The  knowledge  module  consists  of  three 
components.  (1)  First  is  an  option  database  of  context  -  spec i f ic  knowledge. 
In  the  case  of  personnel  selection,  we  would  have  a  database  of  individuals 
(job  applicants)  and  their  relevant  characteristics  (salary  demand,  years 
of  experience,  etc.).  This  part  of  the  knowledge  module  is,  in  fact,  the 
only  aspect  of  the  system  that  is  specific  to  personnel  selection  as 
distinct  from  similar  choice  contexts.  (2)  The  second  component  of  the 
knowledge  module  is  a  preference  model  or  models,  and  an  associated  set  of 
decision  strategies,  which  capture  the  user's  knowledge  of  his  own 
preferences  in  the  problem  domain  and  his  or  her  preferred  method  for  using 
those  preferences  in  the  choice  process.  Preference  models  may  include 


numerical  weights  or  attributes,  intervals  or  ratios  of  weights,  rank 
ordering  of  weights,  cutoffs,  or  direct  evaluations  of  options.  Thus, 
preferences  may  be  specified  by  the  user  with  varying  degrees  of 
"fuzziness"  and  may  be  incomplete  in  some  or  many  respects.  User- se lec ted 
decision  strategies  may  be  based  on  multiattribute  utility,  elimination  by 
aspects,  satisficing,  or  dominance,  and  are  the  means  by  which  options  from 
the  database  are  retained  or  rejected.  (3)  The  third  component  of  the 
knowledge  base  is  information  regarding  the  current  status  of  the  system. 
For  example,  the  knowledge  module  will  keep  track  of  which  options  in  the 
database  are  "live"  and  which  have  been  eliminated  from  consideration  (via 
application  of  a  preference  model  by  use  of  the  Decide  module).  In 
addition,  the  user  may  have  indicated  that,  although  the  model  is  specified 
in  terms  of  a  large  number  of  attributes,  (s)he  wishes  to  evaluate  options 
based  only  on  some  subset  of  attributes. 

A. 2. 2  Adjust  module.  This  module,  which  acts  on  the  first  two  components 
of  the  database,  has  two  major  functions.  The  first  function  is  to  allow 
the  user  to  make  changes  in  the  option  database  as  more  information  is 
acquired  (e.g.,  a  candidate  is  no  longer  available,  the  performance  of  an 
option  on  an  evaluative  dimension  needs  to  be  changed  in  the  light  of  new 
information).  The  second  (and  most  important  for  a  personalized  decision 
aid)  function  is  to  create  or  change  the  preference  models  by  means  of 
which  the  options  in  the  database  are  to  be  evaluated.  It  is  this  function 
that  can  most  significantly  be  tailored  to  individual  cognitive  styles. 

The  adjust  module  allow's  the  user  to  specify  preferences  in  three  basically 
different  ways:  (1)  compensatory  knowledge  about  the  relative  importance 
(and  tradeoffs)  among  evaluative  criteria;  (2)  cutoff  levels,  i.c.,  non¬ 
compensatory  goals,  on  attributes  such  that  candidates  not  meeting  the 
cutoffs  are  rejected;  and  (3)  "bootstrapping"  or  "policy  capture"  judgments 
which  express  the  values  of  options  directly.  These  three  modes  correspond 
to  differences  in  the  degree  to  which  users  prefer  analytic  v<  rsus 
intuitive  and  concrete  approaches  to  choice.  In  addition,  as  noted, 
analytic  knowledge  in  mode  (1)  may  be  expressed  to  virtually  any  d* gr<  t  of 
precision/imprecision  or  comp  1 e t eness/i ncomp 1 e t ones s . 
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4.  2. 3  Decide  module.  The  function  of  this  module  is  to  apply  a  user- 
preferred  decision  strategy,  and  a  user-defined  preference  model  to  the 
database  of  options,  and  evaluate  and  display  the  results.  If  a  full 
multiattribute  utility  model  has  been  specified  (i.e.,  a  complete  set  of 
numerical  weights  on  all  attributes),  the  system  can  display  the  most 
preferred  al ternat ive ( s ) .  If  the  model  is  only  partially  specified  (i.e., 
user  inputs  of  orderings,  intervals  and/or  ratios  of  weights  fall  short  of 
entailing  exact  values  of  weights),  the  system  displays  those  implications 
that  can  be  drawn  from  the  information  the  system  has.  For  example,  the 
system  can  compute  the  feasible  alternatives,  i.e.,  those  options  that 
cannot  be  eliminated  based  on  the  available  knowledge  of  the  user’s 
preferences.  In  addition,  the  system  can  perform  a  type  of  dominance 
testing;  that  is,  given  one  alternative  (hypothetical  or  from  the 
database) ,  it  can  compute  which  alternatives  are  known  to  be  at  least  as 
good.  Finally,  the  system  can  perform  elimination  by  aspects,  a  choice 
strategy  requiring  only  an  ordering  on  the  attributes  and  cutoff  levels  for 
each  attribute. 

4.2.4  Select  module .  This  module  allows  the  user  to  select;  (1)  aspects 
of  the  preference  model  for  display  or  adjustment,  (2)  information  about 
options  for  display  or  adjustment,  and  (3)  a  subset  of  the  current 
preference  model  for  application  in  choice. 

4.2.3  Alert  module .  The  function  of  this  module  is  to  notify  the  user  of 
relevant  changes  in  knowledge  about  options.  This  module  is  of  primary 
importance  in  real-time  systems  in  which  the  system  must  assimilate 
information  about  the  outside  world  faster  than  it  can  be  attended  to  by 
the  user.  Such  a  system  must  help  the  user  to  cope  with  "ini ormnt i on 
overload"  bv  alerting  the  user  'o  possibly  useful  new  information. 

F.ven  in  the  present  context  .  however,  informat  ion  overload  mav  plague  the 
user  of  a  large  database.  If  n> v  opt  inns  a  re  incorporated  into  the 
database,  or  changes  occur  in  the  data  r<  .'.aiding  old  options,  then  the  use  i 
might  fie  alerted  it  land  ot  lv  it  .1  those  changes  have  implications  for  the 


A  second  mode  of  alerting  involves  use r - spec  if ied  alerts:  for  example,  the 
system  can  be  told  to  provide  an  alert  if  a  candidate  shows  up  with  four 
years  of  programming  experience. 

4.2.6  Advisory  module.  This  module  functions  as  an  overseer  of  the  other 
modules,  checking  when  a  preferred  user  decision  strategy  may  result  in 
significantly  suboptimal  choices.  (1)  Adjust:  For  example,  users  would  be 
prompted  when  inputs  provided  under  the  Adjust  module  are  significantly 
inconsistent,  and  v/ays  of  resolving  the  inconsistency  would  be  suggested. 

If  the  information  provided  by  a  user  is  too  imprecise  or  incomplete  for 
successful  application  of  the  preferred  decision  strategy,  Advisory  prompts 
would  suggest  additional  inputs  that  are  needed  (e.g.,  the  number  of  live 
options  could  be  narrowed  down  from  100  to  the  desired  10,  if  only  the 
order  of  importance  of  education  and  experience  is  indicated) .  Users  might 
also  be  notified  if  small  changes  in  user  inputs  would  have  a  significant 
impact  on  choice.  (2)  Decide:  in  the  Decide  module,  users  applying 
elrmination-by-aspects  or  satisficing  would  be  prompted  to  reconsider  an 
option  which  has  been  discarded  because  it  fails  to  meet  a  cutoff  on  one 
attribute,  but  is  outstanding  in  other  respects.  Conversely,  the  user 
would  be  notified  if  an  option  which  is  accepted,  because  it  clears  all 
cutoffs,  is  in  fact  dominated  (i.e.,  there  is  an  option  set  which  is  at 
least  as  good  as  that  option  on  all  dimensions  and  better  on  at  least  one 
dimension).  (3)  Select:  Users  would  be  prompted  when  user- selected 
attributes  fail  to  capture  significant  variance  in  the  option  database. 

(4)  Alert:  Users  would  be  prompted  when  user-defined  alerting  criteria 
fail  to  capture  significant  aspects  of  the  current  preference  model. 

In  all  cases,  users  would  control  the  degree  of  significant  of  the  problem 
that  is  to  trigger  an  advisory  prompt.  Moreover,  users  are  free  to  accept 
or  reject  any  advice  that  is  offered. 

'  3  Current  Status 

This  section  describes  in  somewhat  more  detail  selected  functions  in  the 
personal i red  evaluation  system  that  are  currently  implemented  or  planned. 


$ 


4  .  3 . 1 . 1  Problem  structuring- -This  function  allows  initial  structuring  of 
the  problem,  i.e.,  identification  of  the  options  to  be  considered  and  the 
attributes  on  which  they  are  to  be  evaluated. 

4 . 3 . 1 . 2  Entering  comnensatorv/tradeoff  information  on  attribute  weirhts - 
The  user  can  enter  compensatory/tradeoff  information  on  the  relative 
importance  of  attribute  weights  in  any  one  of  three  ways  (or  in  any 
combination  of  the  three): 


o  The  Weights  Screen  -  On  the  Weights  Screen  (Figure  3),  the  user 

can  enter  upper  and  lover  bounds  for  each  attribute  weight.  By 
setting  the  upper  and  lower  bounds  equal,  a  precise  weight  may 
be  entered.  The  user  also  sees  the  bounds  implied  by  all  other 
judgments  (s)he  has  made  on  this  and  any  other  screens.  Figure 
3  .shows  a  sample  Weights  screen  display,  on  which  the  user  has 
indicated,  for  example,  that  the  weight  for  the  first  attribute 
(Experience)  should  be  between  15  and  35  on  a  scale  from  0  to 
100,  with  the  other  attributes  weighted  as  shown. 

o  The  Tradeoffs  Screen  -  The  Tradeoffs  Screen  (Figure  4)  allows 

comparisons  of  relative  magnitudes  of  attribute  weights.  One  of 
the  attributes  may  be  specified  as  the  standard,  and  other 
attribute  weights  are  assessed  relative  to  it.  In  the  sample 
screen  of  Figure  4,  the  decision  maker  has  indicated  that  the 
Education  attribute  should  be  given  weight  between  70%  to  90% 
of  that  of  the  Experience  attribute.  The  information  is  coded 
internally  into  linear  inequality  constraints  on  the 
attributes.  Again,  the  decision  maker  can  see  the  implications 
of  all  other  inputs  to  the  system  (on  this  or  any  other  screen) 
as  they  relate  to  relative  magnitudes  of  attribute  weights. 

o  The  Rank  Order  Screen  -  On  this  screen  (Figure  5)  the  user  can 

enter  information  merely  about  the  order  of  importance  of 
attribute  weights  (e.g.,  that  the  first  attribute  should  have 
higher  weight  than  the  second  without  saying  any  more  about 
what  those  weights  are).  Once  again,  the  implications  of  all 
user  inputs  for  rank  order  of  weights  is  displayed. 
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Each  of  these  screens  allows  a  user  to  think  about  his  or  her  preferences 
in  a  different  way.  Yet  they  are  highly  integrated,  since  the  implications 
of  all  currently  available  information  (entered  from  any  of  the  above 
screens,  or  from  the  bootstrapping  screen  described  below)  can  he  displayed 
on  each  screen.  The  mechanism  for  computing  implications  across  screens  is 
linear  programming. 

The  entering  of  information  on  each  of  these  screens  corresponds  to 
building  or  changing  a  preference  model. 

4. 3. 1.3  Cutoffs  on  attribute  scores--On  the  Cutoffs  screen  (Figure  6),  the 
user  can  indicate  minimal  and/or  maximal  acceptable  levels  for  any  attri¬ 
bute,  so  that  alternatives  not  meeting  the  cutoff  are  eliminated  from 
consideration.  Cutoffs  can  also  be  provided  using  the  Elimination-by- 
Aspects  function,  described  below. 

4. 3. 1.4  Bootstrapping  (planned) - -On  the  bootstrapping  screen,  the  user 
makes  direct  judgr.  ants  about  alternatives,  either  real  or  hypothetical. 

The  user  may  place  bounds  on  the  score  of  the  alternative  (e.g.,  "between 
65  and  80  on  a  scale  of  100”)  or  may  make  a  direct  comparison  between  two 
alternatives  (e.g.,  "Jones  would  score  better  than  Smith").  As  for  the 
above  screens,  bootstrapping  information  can  be  encoded  as  linear 
inequalities  on  attribute  weights. 

Bootstrapping  is  an  indirect  way  of  giving  the  system  information  on 
attribute  weights.  Hence,  the  Bootstrapping  screen  may  be  viewed  as  part 
of  the  Adjust  module,  and  implications  of  the  Bootstrapping  judgments  are. 
stored  as  part  of  the  Knowledge  module. 


4.3.2  Decide .  By  pointing  to  "Maximize"  and  clicking  the  mouse  key,  the 
user  instructs  the  system  to  apply  whatever  knowledge  about  his  preferences 
the  system  has  gathered  to  the  current  set  of  options.  Two  settings  are 
provided  which  influence  the  performance  of  this  function:  Needed" 

indicates  the  number  of  options  which  the  user  ultimatelv  desires  to 
choose;  it  mar  range  from  1  to  the  number  of  items  in  the  dat  abase  (a 
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trivial  choice  problem!).  The  first  time  "Maximize"  is  selected,  the 
preference  model  is  applied  to  the  entire  database  of  options.  The 
application  of  that  model  is  likoly  to  result  in  a  smaller,  weeded  down 
subset  of  options  which  remains  feasible  or  "live"  (i.e.,  a  set  of  options 
which  is  known  to  contain  the  desired  choices,  but  within  which  further 
discriminations  cannot  be  made  based  on  present  inputs).  The  user  may,  if 
he  wishes,  return  to  the  Adjust,  screens,  provide  further  inputs  (or  revise 
old  ones),  and  then  select  "Maximize"  again.  Each  subsequent  selection  oi 
"Maximize”  operates  on  the  currently  prevailing  set  of  live  options, 
further  narrowing  it  down.  The  user  has  the  option,  however,  of  resetting 
this  process  by  selecting  "Try  all  options."  In  that  case,  the  next  use  of 
"Maximize"  operates  on  the  original,  full  database. 

By  these  simple  commands,  the  user  can  implement  any  of  a  variety  of  choice 
strategies.  For  example,  by  setting  cutoffs  on  additional  dimensions 
between  each  use  of  "Maximize,"  he  may  successively  eliminate  options, 
through  an  elimination-by-aspects  strategy.  Any  compensatory  information 
will  also  be  used  directly  by  the  system  to  eliminate  options.  The  result 
of  "Maximize"  is  always  the  set  of  feasible  options,  i.e.,  those  that  could 
be  optimal  given  current  information. 

4.3.3  Guides .  In  addition  to  these  basic  functions,  a  variety  of  guides 
and  special  screens  support  the  decision  process.  These  involve  functions 
not  only  from  the  Decide  module,  but  also  from  Adjust,  Select,  and 
Advisory,  orchestrated  in  a  user-friendly  dialogue  that  reflects  typical 
decision  making  patterns. 

4. 3. 3.1  El iminate- -For  example,  instead  of  performing  elimination-by¬ 
aspects  by  means  of  elementary  commands  (as  described  above),  the  user  can 
receive  additional  guidance  through  "Eliminate."  This  guide  directs  the 
user,  in  a  highly  flexible  manner,  through  the  following  steps: 


(1)  Selection  of  attributes  for  inclusion  in  the  choice  process  (bv 
graphical  pointing)  (Figure  7); 


(2)  use  of  ordering  screen  to  rank  order  the  selected  attributes, 
with  prompts  to  elicit  from  the  user  a  single  connected 
ordering  (Figure  8); 

(3)  use  of  cutoff  screen  to  elicit  cutoffs  for  the  selected 
attributes  (Figure  9); 

(4)  display  of  results  on  the  EBA  screen  (Figure  10). 

These  steps  are  hy  no  means  rigid.  For  example,  if  he  or  she  is  satisfied 
with  the  already  existing  attribute  selection,  ordering,  and  cutoff 
specification ,  the  user  may  proceed  directly  to  step  (4).  At  any  time 
thereafter,  the  user  may  return  to  any  previous  step,  make  changes,  and 
observe  the  results. 

The  EBA  screen  (Figure  10)  functions  as  an  elimination-by-aspects 
"spreadsheet."  It  displays  the  selected  attributes  in  order  of  importance, 
the  lower  and/or  upper  cutoffs,  and  the  number  of  surviving  options  at  each 
stage  in  the  process  (i.e.,  after  each  additional  set  of  cutoffs  has  been 
applied).  In  addition,  an  advisory  prompt  is  provided  which  notifies  the 
user  if  there  are  promising  options  which  have  been  rejected.  These  are 
options  which  fall  outside  the  cutoffs  on  an  attribute,  but  score  very  well 
on  other  dimensions.  In  addition,  the  user  is  told  by  how  much  the 
specified  cutoffs  would  have  to  be  stretched  to  include  these  rejected  (but 
promising)  options. 

The  user  can  modify  any  cutoff  directly  on  this  screen,  and  the  EBA  will  be 
recomputed,  showing  which  alternatives  are  thereby  excluded  or  re-included. 
By  use  of  the  mouse  and  the  VIEW  option,  the  user  can  "zoom"  in  on  the 
details  of  any  part  of  the  EBA  screen.  For  example,  the  user  can  view  all 
database  options  in  the  vicinity  of  the  lower  or  upper  cutoff  on  any 
attribute,  or  he  can  view  the  set  of  promising  but  rejected  options.  The 
reasons  for  considering  an  option  "promising"  are  also  indicated.  From  the 
zoomed-in  point,  the  user  can  scroll  anywhere  else  in  the  database  of 
options.  The  VIEW  screen  can  also  be  used  to  modify  cutoffs  in  a  way  that 
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differs  significantly  from  the  CUTOFFS  screen  or  the  EBA  screen.  The  user 
may  graphically  move  upper  and  lower  cutoffs  against  the  context  of  actual 
options  to  be  included  or  excluded,  rather  than  merely  against  an  abstract 
numerical  scale. 

A. 3. 3.2  Justify  (partially  i mpl emented) - -The  Justify  screen  (Figure  11) 
provides  users  with  a  comprehensive  evaluative  overview  of  selected 
options.  The  objective  is  to  support  the  final  stages  of  a  decision 
process,  in  which  users  seek  to  examine  candidate  choices  in  detail,  to 
look  for  previously  overlooked  weak  points,  and  to  construct  a 
j us t i f i ca t i on  for  the  choice.  The  Justify  screen  for  a  given  option 
displays  the  score  of  that  option  on  each  attribute,  the  best  score  of  any 
option  on  each  attribute,  and  the  difference  (in  standard  deviations) 
between  the  current  option's  score  and  the  best  score;  it  also  shows  how 
many  options  scored  better  than  the  present  one  on  each  attribute  and  what 
percentage  of  the  total  database  they  represent.  Finally,  it  indicates  if 
the  present  option  is  poor  or  outstanding  on  any  particular  dimension. 

A  VIEW  option  is  associated  with  Justify  which  enables  users  to  "zoom"  in 
on  and  examine  in  detail  the  options  which  are  superior  to  the  given  option 
in  any  particular  dimension. 

For  some  decision  makers,  the  Justify  screen  might  play  a  central  role  in 
decision  making.  The  process  of  justifying  a  choice  may  lead  to 
reconsideration  and  revision  of  previous  preference  judgments.  Montgomery 
(1983)  has  argued  that  decision  making  in  general  is  a  search  for  good 
arguments  or  justifications.  A  characteristic  strategy  is  to  accept  an 
option  as  justified  only  if  efforts  to  create  a  dominance  structure  for 
that  option  are  successful,  i.e.,  when  preferences  can  plausibly  be 
represented  in  such  a  way  that  the  option  appears  as  good  or  better  than 
other  options  on  all  dimensions.  A  stronger  justification  occurs  when  the 
preferred  option  is  shown  to  be  uniquely  outstanding  on  at  least  one 
dimension.  The  Justify  screen  enables  users  to  ascertain  quickly  whether 
requirements  -'f  this  sort  are  satisfied.  if  not,  users  iii.iv  alter  the 
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representation  of  their  preference  model  until  dominance  applies,  e.g.,  by 
combining  attributes  (via  the  problem  structuring  screen) ,  by  reducing  the 
importance  of  attributes  on  which  the  preferred  option  is  poor,  or  by 
reconsidering  the  assessment  of  the  performance  of  one  or  more  options  on 
an  attribute. 

Advisory  prompts  will  notify  users  when  an  option  is  itself  dominated,  or 
is  especially  poor  in  some  dimension.  In  addition,  the  system  will  track 
user  efforts  to  establish  a  dominance  structure,  and  prompt  when 
alterations  in  previous  preferences  are  excessive  according  to  a  user-set 
criterion . 

4. 3. 3. 3  Focus  (planned')  - -Traditional  multiattribute  utility  analysis 
requires  precise  numerical  assessments  of  the  relative  importance  of  all 
evaluative  dimensions.  Both  ELIMINATE  and  JUSTIFY  simplify  the  choice 
process  by  enabling  the  user  to  employ  simple  non-compensatory  judgments, 
i.e.,  comparisons  within  (rather  than  across)  attributes.  An  ordering  of 
attributes  by  importance  is  the  only  across-attribute  information  required. 
The  result  is  a  selection  of  options  that  may,  on  occasion,  overlook 
important  tradeoffs.  Advisory  prompts  are  provided  to  protect  against 
this.  An  alternative  strategy  for  simplifying  choice,  however,  is  to 
retain  the  goal  of  selecting  alternatives  with  the  highest  expected  utility 
(i.e.,  the  highest  overall  score  allowing  for  tradeoffs),  but  to  elicit 
compensatory  assessments  from  users  only  to  the  degree  required  by  the 
particular  choice  problem  at  hand. 

The  FOCUS  guide  steps  the  user  through  the  minimal  set  of  assessments 
required  to  narrow  the  option  set  to  the  number  desired.  FOCUS  begins  with 
the  weakest  possible  test,  dominance,  to  determine  if  that  is  sufficient  to 
obtain  the  needed  weeding  out.  Only  if  this  fails  does  it  step 
progressively  through  queries  for  stronger  and  stronger  inputs  from  the 
user.  In  all  cases,  it  seeks  out  aspects  of  preference  knowledge  that  are 
likely  to  have  the  greatest  impact  on  narrowing  down  the  option  set  (e.g. , 


Che  order  of  importance  of  attribute  A  and  B;  whether  C  is  at  least  twice 
as  important  as  D;  etc.)-  As  a  result,  the  user's  choice  problem  may  be 
solved  with  the  least  judgment  effort  from  the  user. 

In  addition,  at  any  point,  the  user  has  the  option  of  short-circuiting  this 
process  by  instructing  the  system  to  "extrapolate"  a  set  of  constraints 
that  would  solve  the  problem.  The  system  then  selects  a  set  of  weights 
consistent  with  the  information  thus  far  provided  by  the  user,  and  ranks 
the  alternatives  with  respect  to  those  weights.  A  sensitivity  analysis  is 
also  displayed,  to  permit  the  user  to  assess  how  sensitive  the  ranking  is 
to  the  set  of  weights  chosen.  If  he  desires,  the  user  may  then  provide 
additional  judgments  of  his  own  to  "correct"  the  system's  extrapolation . 

A . 4  Implementation 

This  section  provides  a  summary  of  the  hardware  and  software  approach. 

4.4.1  Hardware .  An  IBM  AT  system  was  used  for  development  of  the  aid. 
Software  produced  operates  on  both  PC-  and  AT-compatible  machines.  These 
are  both  16-bit  microprocessors,  the  PC  having  an  8-bit  data  path  provided 
by  the  Intel  8088  processor,  and  the  AT  having  a  16-bit  data  path  due  to 
its  Intel  80286  processor.  For  development  purposes,  machines  have  been 
equipped  with  floating  point  co-processors  (8087/80287)  to  improve 
computational  speed,  hard  disks,  and  at  least  512KB  of  random  access 
memory.  The  developed  software  does  not  require  the  co-processors  (but 
will  utilize  them  when  available)  and  requires  no  more  than  256KB  of  memory 
(the  minimum  available  on  the  PC  or  AT). 

The  aiding  system,  like  the  submarine  approach/attack  planning  aid 
developed  under  Phase  I  of  this  project,  utilizes  interactive  graphics 
heavily.  We  anticipate  that  the  present  low-resolution  (320  x  200  pixels 
with  4  simultaneous  colors)  video  systems  used  by  personal  computer  owners 
will  be  largely  replaced  within  the  next  two  to  three  years  with  higher 
resolution  systems.  For  this  reason,  we  are  utilizing  the  recently 
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available  IBM  Enhanced  Graphics  Adapter  (640  x  350  pixels  with  16 
simultaneous  colors)  and  an  RGB  color  monitor  of  no  less  than  640  x  350 
pixel  resolution. 

4.4.2  Software  Software  consists  of  the  following: 

o  A  set  of  software  modules  written  in  C  and  comprising  the  user- 

system  interface.  These  provide  the  user  link  to  the  various 
personalizing  modules  (Select,  Planning,  Adjust,  Alert, 
Advisory) . 

o  A  Data  or  Knowledge  module  consists  of  two  components:  (1)  a 

processing  subsystem  written  in  C  containing  the  rules 
appropriate  to  support  evaluations  based  on  a  variety  of 
techniques,  ranging  from  elimination  by  aspects  to 
multiattribute  utility  analysis;  (2)  a  database  management 
subsystem,  written  in  C  and  utilizing  in  addition  components  of 
an  existing  system,  dBASE  III. 

o  A  graphics  interface  module.  Because  of  the  current  lack  of 

standards  for  graphics  software  and  the  variety  of  graphics 
hardware  which  must  be  supported  if  reasonable  market 
penetration  is  to  be  achieved,  all  graphics  functions  are 
performed  by  a  distinct  software  module.  This  will  make  future 
modifications  relatively  simple.  This  is  written  in  C, 
utilizing  a  graphics  system  known  as  "Halo"  and  developed  by 
Media  Cybernetics,  Inc.  Programming  of  this  system  is  somewhat 
similar  to  programming  of  the  Virtual  Device  Interface  (VDI) 
graphics  system  being  released  very  shortly  by  IBM.  The  VDI 
approach  offers  the  capability  of  automatically  utilizing 
whatever  resolution  is  available  on  a  (VDI -supported)  device. 
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5.0  EXPERIMENTAL  RESEARCH  ON  INDIVIDUAL  DIFFERENCES 
IN  DECISION  STRATEGIES 

5 . 1  Method 

A  personalized  decision  aid  may  serve  as  a  productive  testbed  for 
exploration  of  hypotheses  regarding  individual  difference  in  preferred 
decision  making  strategies.  A  full  exploration  of  this  potential  was  not 
possible  within  the  resource  constraints  of  the  present  project;  however,  a 
very  preliminary  study  was  conducted  in  order  to:  a)  test  the  viability  of 
hypotheses  underlying  personalized  and  prescriptive  decision  aiding 
techniques,  b)  illustrate  the  use  of  the  prototype  system  as  an 
experimental  testbed. 

The  following  research  questions  were  addressed: 

o  Do  people  in  fact  use  different  problem  solving  approaches? 

o  Is  an  aid  which  adapts  to  user  differences  preferable  to  an 

inflexible  aid? 

o  Do  different  tasks  induce  different  problem  solving  approaches? 

o  Do  different  cognitive  styles  of  users  induce  different  problem 

solving  approaches? 

These  questions  were  addressed  by:  a)  providing  a  flexible  condition  in 
which  subjects  could  select  their  own  decision-making  strategy  utilizing 
the  prototype  aid;  b)  comparing  conditions  in  which  users  could  flexibly 
select  their  own  decision-making  strategy  with  conditions  in  which  a 
particular  strategy  was  imposed  upon  them;  c)  providing  tasks  which 
differed  in  the  degree  and  nature  of  the  uncertainty  about  values;  d) 
comparing  users  who  were  se  1  f  -  desc  r  ibed  analytics  with  users  who  were  .‘.elf- 
described  intuitives. 


L 


The  study  employed  by  a  3  x  4  x  3  design  with  two  within-subjects  factors: 
task  type  and  decision  strategy,  and  one  be  tween- sub } ects  factor:  cognitive 
style.  Each  subject  played  the  role  of  a  personnel  director  of  a  large 
corporation.  The  subject  was  presented  with  twelve  written  scenarios.  In 
each  scenario,  the  subject  had  to  hire  three  people  out  of  twenty 
candidates  for  a  particular  job. 

Task  types  varied  as  follows: 

o  Uncertainty:  Subjects  were  provided  a  description  of  job 

requirements,  which  was  mis-matched  with  information  about 
candidates  provided  on  their  resumes. 

o  Certainty:  Subjects  were  given  descriptions  of 

of  the  relevant  department  head  in  a  form  which 
information  provided  on  the  resumes. 

o  Conflict:  Subjects  were  given  descriptions  of  the  preferences 
of  two  relevant  department  heads ,  each  of  whom  might  make  use 
of  the  new  employee,  in  a  way  which  matched  resume  information; 
the  preferences  of  the  two  department  heads  were  inconsistent 
with  each  other. 

The  decision  strategy  condition  was  either  flexible  or  inflexible.  If 
inflexible,  subjects  were  told  which  of  three  different  decision  strategies 
to  use  in  each  scenario.  The  possible  strategies  were: 

o  Eliminate:  Assess  cutoffs  (or  minimum  requirements)  on  each 

dimension . 

o  Weights:  Assess  the  relative  importance  of  different 

dimensions . 

o  Tradeoffs:  Assess  ranges  of  possible  relevant  importance  of 


the  preferences 
matched 


dimensions . 


In  the  flexible  condition,  subjects  were  able  to  choose  for  themselves 
which  of  these  strategies  they  would  use  in  each  scenario. 

In  addition,  subjects  were  asked  to  provide  ratings  of  their  own  approach 
to  problem-solving  on  two  scales:  one  reflecting  their  degree  of 
intuitiveness,  and  the  other  reflecting  their  degree  of  analy t ical i ty .  All 
subjects  fell  into  one  of  three  groups:  those  who  rated  themselves  high 
(greater  than  4  on  a  7 -point  scale)  on  intuitivenes-s ,  but  not  on 
analyticality ,  those  who  rated  themselves  high  on  analyticality ,  but  not  on 
intuitiveness,  and  those  who  rated  themselves  high  on  both  dimensions. 

For  each  problem,  the  time  required  to  solve  the  problem,  the  subject's 
subjective  confidence  in  the  solution,  and  thp  subject's  subjective 
satisfaction  with  the  system  on  that  problem,  were  recorded.  In  addition, 
in  the  flexible  conditions,  the  subject's  choice  of  decision  strategy  was 
recorded.  Also,  a  process  trace  of  each  subject's  use  of  the  system  has 
been  stored. 

Predicted  impact  of  task  conditions  on  decision  strategies .  In  the 
uncertainty  condition,  the  department  heads'  preferences  in  terms  of  hiring 
are  not  stated  in  terms  of  the  attributes  which  are  provided  in  the 
resumes.  The  user  must  bridge  this  gap,  i.e.,  he  must  predict  or  infer  the 
causal  impact  of  traits  described  in  the  resume  on  achievement  of  the 
department  head's  goals.  For  example,  if  the  department  head's  preferences 
include  that  the  new  employee  write  well  and  work  well  with  numbers,  then 
the  subject  might  infer  that  a  high  score  on  the  education  test,  as 
described  in  the  resume,  will  be  correlated  with  the  achievement  of  those 
goals.  Similarly,  if  the  department  head's  preferences  include  handling 
unexpected  situations  or  crises,  then  the  subject  might  predict  that  a  job 
candidate  with  many  years  experience  would  do  well  on  those  objectives. 

This  type  of  causal  modeling,  rudimentary  though  it  is,  is  not  very  well 
supported  by  decision  strategies  that  require  judgments  of  the  relative 
importance  of  different  dimensions.  It  is  not  the  relative  importance  of 
education,  and  experience,  that  most  concerns  the  subject  in  this 
condition.  It  is  rather  what  level  of  education  is  required  to  ensure  a 
high  likelihood  that  the  candidate  will  write  we.l  1  and  work  well  with 
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numbers.  Similarly.  the  sub  ter  would  be  concerned  with  what  level  of 
experience  is  requited  * < >  ensure  a  high  likelihood  that  the  candidate  will 
handle  un-  >  p*-c  r  *-<;  .  us  <  ■  r  <  rises  adequately.  Assessments  of  relat  i  ve 

importance  would  s  'ear's  at  t  ent ion  on  the  required  causal 

mode  1  inf, .  causa.  .  :  .•  •  >•  '  k  •  <  v ledge  appears  to  he  represented  at  the 

level  of  ccm.'  :  e '  t  >  t  .  •:  Tet-.s  :  or.s  .  rat  her  than  at  the  more  abstract 

level  of  " mu  1  ’  :  .,•••;•  . •  •  •  "  It.  or.trast  .  assessment  of  goals  on 

individual  <i  i  o «  r  .  •  -•  '.  .  .  :  •  :  •  he  causa!  process  required  in  this 

task.  Moreover.  .«•.  ar..  •  •  •{  :  nq  t  „nce  w«  ights  will  he  unne*  essarv, 

unless  all  goals  -..tor."'  ••  s  iiru!  :  atieous  1  v  achieved  .  Our  hypothesis  ,  then  , 

is  that  in  the  uncet  :  a  it;*  v  "i  il:  •  in  ,  1 1,.  re  will  he  a  greater  tendency  to 
“  the  el  ini::. it  e  de.  i  s  i  on  s'taterv  as  opposed  to  either  we  s  or 
tradeoffs . 

In  the  conflict  condition,  subjects  are  presented  with  inconsistent  sets  of 
objectives,  representing  the  goals  of  different  departments.  Their  cliche 
must  somehow  reconcile  these.  In  this  condition,  it  might,  be  difficult  for 
subjects  to  assess  unique  cutoffs,  or  unique  importance  weights,  for  each 
dimension.  The  tradeoffs  strategy,  however,  enables  the  subjects  to 
specify  ranges  of  relative  importance  for  the  dimensions,  thus  capturing, 
the  ambiguity  in  preferences  of  the  two  department  heads.  Moreover,  the 
tradeoff  strategy  lets  the  subject  know  immediately  how  important  conflict 
is;  if  the  total  set  of  job  candidates  is  whittled  down  to  the  required 
three,  despite  the  ambiguity,  then  there  is  no  need  for  the  subject  to 
invest  further  effort  in  resolving  the  conflict.  Our  hypothesis  in  this 
condition,  then,  is  that  the  subjects  will  make  greater  use  of  the  tradeoff 
strategy . 

In  certainty  conditions,  subjects  are  given  qualitative  descriptions  of  t he 
importance  of  the  objectives  in  terms  which  are  directly  related  to  the 
attributes  provided  in  the  resumes.  It  is  plausible  to  suppose  that  these 
qualitative  descriptions  will  direct,  the  subject's  attention  to  t  la- 
relative  importance  of  the  objectives.  Our  hypothesis  in  this  condition 
then,  is  that  the  subjects  will  make  greater  use  of  tin  weights  decision 
strategy . 


Figure  12  summarizes  the  three  hypotheses  just  described. 

5 . 2  Results 

The  most  fundamental  principle  of  personalized  and  prescriptive  decision 
aiding  is  that  different  decision  strategies  will  be  used  as  a  function 
either  of  the  individual  or  of  the  task.  This  hypothesis  was  dramatically 
confirmed.  Figure  13  shows  that  all  three  strategies  were  utilized  in  the 
course  of  the  study.  Eliminate  and  weights  were  used  approximately  equally 
often,  while  tradeoffs  was  utilized  somewhat  less  frequently.  A  given 
subject,  moreover,  did  not  focus  exclusively  on  a  single  strategy.  As 
shown  in  Figure  13,  ten  of  the  thirteen  subjects  utilized  two  strategies  in 
the  course  of  the  study,  while  only  two  of  the  subjects  utilized  a  single 
strategy  throughout. 

Do  subjects  prefer  decision  aids  which  adapt  to  their  preferences  in 
decision  strategy?  The  data  suggests  that  they  do.  Subjects  experienced 
more  confidence  in  solutions,  and  more  satisfaction  with  the  system,  with 
the  flexible  conditions,  as  compared  with  the  non-flexible  conditions 
(Figure  14) . 

It  should  be  pointed  out,  however,  that  flexibility  comes  with  a  price. 
Subjects  took  more  time  in  the  flexible  conditions,  as  compared  with  non- 
flexible  conditions,  as  shown  in  Figure  15.  Increased  time  on  the  problem 
might  be  regarded  as  a  potential  explanation  of  greater  satisfaction  and 
confidence  in  the  flexible  condition.  However,  it  is  not  the  case  that 
subjects  experienced  greater  satisfaction  and  confidence  in  general  when 
thev  spent  more  time  on  the  problem.  Time  was  inversely  correlated  with 
satisfaction  and  confidence  within  both  flexible  and  inflexible  conditions. 

We  turn  now  to  the  impact  of  task  type  and  cognitive  style  on  preference 
and  selection  of  decision  strategies.  Figure  16  suggests  that  task  type 
influences  the  confidence  and  satisfaction  obtained  from  a  given  decision 
s'  r.i'fi'.v,  bit  that  the  effect  is  not  as  clear  as  predicted  in  Figure  12. 
Note  f;t'st  the  tiie  (-•■rtaintv  condition,  as  expected,  produces  more 
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Figure  12:  Summary  of  Hypotheses 


IN  THE  FLEXIBLE  CONDITION,  ALMOST  ALL 
SUBJECTS  USED  MORE  THAN  ONE 
DECISION  STRATEGY 


§  Subjects 


10 


§  Strategies  Used 


ALL  STRATEGIES  WERE  USED 


§  Subjects 
Using 
Strategy 


Figure  13:  Number  of  Strategies  Used 


CONFIDENCE 


SATISFACTION 


FLEXI3LE 

NON-FLEXIBLE 


UNCERT. 

CERT. 

CONFL 

UNCERT. 

CERT. 

CONFL. 

80 

85 

76 

80 

86 

79 

75 

83 

76 

76 

82 

73 

BUT  SUBJECTS  ' 
THAN  NON 

TOOK  MORE  TIME  IN  FLEXIBLE 
l-FLEXIBLE  CONDITIONS 

MINUTES 

UNCERT. 

CERT. 

CONFL. 

FLEXIBLE 

16.22 

12.17 

15.53 

NON-FLEXIBLE 

1 1.46 

10.87 

13.31 

NOTE:  TIME  WAS  INVERSELY  CORRELATED 
WITH  SATISFACTION  AND  CONFIDENCE, 
WITHIN  BOTH  FLEXIBLE  AND  NON- 
FLEXIBLE  CONDITIONS  (ONE  EXCEPTION: 
NON-FLEXIBLE/TRADEOFFS). 

(SO  TIME  DIFFERENCE  DOES  NOT 
ACCOUNT  FOR  DIFFERENCE  IN 
SATISFACTION  AND  CONFIDENCE  BETWEEN 
FLEXIBLE  AND  NON-FLEX1BLE  CONDITIONS.) 


Fiszuri'  1  ) :  Solution  Tir.o  Tudor  l'lo>:  ibl  o  vs  .  Non-!- 1  r>:  ;  h  1 1 
Cond i t i ons 


CONFIDENCE  IN  SOLUTIONS  AND  SATISFACTION 
WITH  SYSTEM  TENDED  TO  VARY  WITH  DECISION 
STRATEGY  (FOR  A  GIVEN  PROBLEM  TYPE) 


NON-FLEXIBLE 

CONFIDENCE 

SATISFACTION 

UNCERT. 

CERT. 

CO  NFL 

UNCERT. 

CERT. 

CONFL 

ELIMINATE 

77  * 

86 

74 

77  * 

84 

72 

WEIGHTS 

77 

86* 

74 

82 

85* 

72 

TRADEOFFS 

70 

78 

78* 

68 

78 

74* 

*  =  STRATEGY  PREDICTED  BEST  FOR  A  GIVEN  PROBLEM  TYPE 

COMPARING  PREDICTED  BEST  WITH  OTHER  STRATEGIES: 


NON-FLEXIBLE 

CONFIDENCE 

SATISFACTION 

PREDICTED 

80 

79 

NON-PREDICTED 

77 

76 

NOTE:  ELIMINATE  AND  WEIGHTS  WERE  APROXIMATELY  EQUAL 

ACROSS  ALL  PROBLEM  TYPES. 
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confidence  and  satisfaction  than  either  the  uncertainty  or  the  conflict 
conditions,  regardless  of  the  decision  strategy  that  is  used.  There  is 
however,  no  pronounced  advantage  of  any  one  decision  strategy  across  all 
task  types.  Rather,  eliminaCe  and  weights  strategies  appear  to  have  an 
advantage  in  the  uncertainty  and  certainty  conditions,  while  tradeoffs  has 
an  advantage  in  the  conflict  condition.  Thus  it  appears  that  some  of  the 
subjects  were  utilizing  the  tradeoffs  strategy  in  the  predicted  manner: 
i.e.,  to  represent  the  ambiguity  in  preference  caused  by  the  conflict  of 
the  two  department  heads.  The  results  also  suggest  that  the  tradeoffs 
strategy  was  especially  bad  in  the  uncertainty  condition.  This  is 
consistent  with  our  prediction  that  the  uncertainty  condition  requires 
causal  modeling  which  clashes  with  the  requirement  to  carefully  assess 
relative  importance.  The  tradeoffs  strategy  requires  a  great  deal  more 
effort  and  more  assessments  of  relative  importance  than  the  weights 
strategy.  Contrary  to  the  prediction,  however,  there  was  no  advantage  of 
eliminate  over  weights  in  the  uncertainty  condition.  There  was  no  clear 
pattern  of  effects  of  task  type  on  the  choice  of  decision  strategy  in  the 
flexible  condition. 

In  contrast  to  task  type,  there  was  a  clear-cut  influence  of  user  cognitive 
style  on  the  choice  of  a  decision  strategy  in  the  flexible  condition.  As 
shown  in  Figure  17,  the  eliminate  strategy  was  chosen  far  more  frequently 
than  the  other  strategies  by  self -described  intuitive  subjects.  The 
tradeoffs  strategy  was  more  frequently  chosen  by  self -described  analytic 
subjects.  (It  is  of  incidental  interest  to  notice  that  subjects  who 
described  themselves  both  as  intuitive  and  analytic  had  a  pattern  of 
preference  more  closely  matching  that  of  the  intuitive  subjects.)  These 
results  are  compatible  with  the  idea  that  reasoning  concretely  in  terms  of 
specific  goals  on  individual  dimensions  corresponds  to  an  intuitive 
cognitive  style,  while  comparing  the  relative  importance  of  different 
dimensions  corresponds  to  an  analytic,  or  more  abstract  cognitive  style. 


CHOICE  OF  STRATEGY  IN  FLEXIBLE  CONDITION 
WAS  HEAVILY  INFLUENCED  BY  WHETHER  SUBJECT 

WAS  INTUITIVE  OR  ANALYTIC. 


§  PROBLEMS 

INTUITIVE 

ANALYTIC 

EQUAL 

ELIMINATE 

9 

2 

7 

WEIGHTS 

5 

4 

3 

TRADEOFFS 

1 

6 

2 

n=5 

n=4 

n  =  4 

ELIMINATE  WAS  FAVORED  BY  INTUITIVE  SUBJECTS 

TRADEOFFS  WAS  FAVORED  BY  ANALYTIC  SUBJECTS 


5 . 3  Conclusions 


These  results  can,  at  best,  be  regarded  as  highly  preliminary  and 
tentative.  However,  certain  results  stand  out  clearly  and  deserve  further 
exploration.  First,  individual  decision  makers  differ  both  among 
themselves  and  from  task  to  task  in  the  decision  strategy  which  they 
prefer.  Second,  a  personalized  aid  which  facilitates  different  approaches 
to  solving  problems  leads  to  greater  confidence  and  more  satisfaction  with 
the  system.  Third,  it  may  be  possible  to  predict  user  preferences  among 
decision  strategies  by  user  self-assessments  of  cognitive  style.  Finally, 
and  perhaps  most  importantly,  the  data  suggests  that  the  decision  strategy 
options  offered  to  users  by  this  aid  are  meaningful  ones,  and  that  this 
system  may  have  considerable  utility  as  an  experimental  testbed  for 
additional  research. 
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6.0  CONCLUSION 


The  work  reported  here  has  demonstrated  that  principles  of  personalized  and 
prescriptive  decision  aiding,  originally  developed  in  the  context  of 
submarine  command  and  control,  can  be  generalized  successfully  to  a  quite 
different  decision  making  context.  Submarine  attack  planning  and  personnel 
evaluation  differ  in  degree  of  time  stress,  in  the  relative  importance  of 
inference  and  choice,  in  the  availability  of  objective  measures  of  success, 
in  the  organizational  role  of  the  aid  and  the  user,  and  in  numerous  other 
ways.  Yet  each  of  these  contexts  poses  a  similar  requirement  for  decision 
support  that  is  tailored  to  individual  styles  of  problem  solving  and 
decision  making  and  which  provides,  at  the  same  time,  prescriptive  guidance 
and  advice.  A  common  set  of  cognitive  interface  modules  has  been  found  to 
satisfy  this  need  in  each  case. 
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